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A series of experiments investigated the role 
prior knowledge in tasks involving the operation of equipment 
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written instructions. The experiments covered two situations 
first, the prior knowledge was already possessed by the subjects 
before the experiments. The studies involved comprehension and memory 
of technical prose, expertise in descriptions of familiar and 
unfamiliar pieces of equipment, and* expert ise effects in following «• 
instructions that differed in organization. In the second Situation, 
the prior knowledge was provided as part of the training involved in 
the experiments. These studies concerned the role of knowledge of how 
a system works and transfer of training from previously learned 
operating procedures to new procedures. The results support 
methodological,^ theoretical and practical conclusions. 
Methodologically, many traditional pfose recall paradigms should be 
used with caution in the investigation of prior knowledge, and ' 
careful attention should be paid to the relationship between the 
Knowledge being supplied to the subject and the exact tasks that the 
subject is expected to perform. Theoretically, the results support 
what is perhaps becoming the consensus model of cognitive 
architecture, namely the ACT class of theories described in ' 
Anderson's most "recent textbook, the "Architecture of Cognitiony , 
(1983). Practically, the results provide a geod foundation for/fu* 
applied research on the arrangement, sequence and content of / 
instructional materials. (FL> ' // 
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■This final report surmna r i zeif a set of results' on the role 
of prior knowledge in how peool e operate electronic equipment' 
from written instructions: These reslil^ts cover two situations 
I n\ the first, the prior knowledge is pojsesfed by subj'ects 
pri»or to the experiment. These studies^ i n vo i ved t compret>ens ion- 
arid memory of technical text, expertise in descriptions/ of 
faml'lVar and unfamiliar pieces of equipment, and e>xo^rtise 
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effects in following instructions that differ in organization. 
In the second .s i.tua.tion , the prior knowledge was provided -as. 

'• part: of the training involved in the experiments. These 
studies concerned the jro.le.o.f knowledge of how .a system works, * 
and transfer .o7 training from previously-learned operating 
procedures to new procedures.' Simulation mode 1 s. v/ere. constructed 
and compared in detail to the "dajta., yielding significant 
•theoretical cone 1 us i 6ns* about the/mechanisms involved in the 
effective use. of prior knowledge. ' The work has considerable 
practical significance for the design and evaluation of materials 
and documentation for equipment training and maintenance. 
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ABSTRACT • ••• 4 . 

v This final report summarizes a set Of results on the role q'f\ 
prior knowledge .-in, how . people operate electronic equipment from 
written ihstrifct ions. Thes'e results cover two situations:' In 
t*w first/)', the pri'or knowledge is possessed by subjects prior to 
the experiment. These studies involved comprehension and memory 
of technical text,' expertise in descriptions of familiar and 
unfamiliar., pieces of. equipment, and expertise . effects in. 
following . institutions that differ 1 in- organization. .,-Jn the 
second, situation "the prior knowledge- was provided as part of the, 
training involved ■■ in the experiments. These studies concerned 
the role of '.knowledge of how a system Works, and transfer 
of training .from previously-learned operating procedures- to . new 
procedures. Simulation models were constructed 1 ' and compared - in 
detail to the data, - yielding .significant theoretical conclusions- 
about the mechanisms involved, in the- effective use of prior* 
.knowledge. The' -work has considerable practical 'significance for 
the design and evaluation- of material's and ' documentation for 
equipment training and maintenance. ; . , ' 
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•.The; Role of Prfcor . Knowledge in Opje rating ... ,.;v- ' v\ ; - ' 
'•• > Equipment frdm Written instructions ■• ' «• * v •> , 

, . . . • Dav.iiJ -U.. Kierae - . : !/;.;/•'' ' ■ . '. 

. . • :•- •"- I' . Unlver.sity\p'f Mi'cjj^g'an' - - ii- ; ■'■' 

• ■ ':''"'... '. I N T R.oibtrc T 1 ON-. " '' 'V; v';.' ; 

■■ • • • ,■ . »• ; * ' . ■' ■ v «. • i ■, 

• ( ' * " - ." ■ < 

' \^ ' » \ 1 ,« t ^ . . . ■ . 

. The goal- of. this project was the stijidy.- of: .the rol^tof ; 'priQr 
'knowledge in tasks^v involving ,'$he ~ J ope rati oh of . equipment, from 
written instructions^ The ,Worik has been done/ in '/two* categories: 
In the first', prior ' knowledge tias ' hot ' manipulated.^ -' rather.- the 
prior knowledge * under . investigation was the knowledge, possessed' 
by trie perso'n prior ' to * :be$ing, in the experimental. 'situation. 
This pro'ject began, by ., investigating the roleVof.', 'prior knowledge 
in the, comprehension of simple 'technical 'prosed since,.' a large, 
body of reaearch ' has- > been d;ohe,» j ' on- how- people learn information- 
from Written prose / "arid .the i ''-previous /.ONfr projetTt • focuss'ed"^n. 
- comprehension '■•>.&$; de^erlp^ive technical -t.ext: .* • Tfyis work wa^s 
•.followed" by a 's«t .of ..• studies on. ,>hat' v exp^rt^ ' .know about 
electronic equipment compare^ to- ft on- experts; * alori^ '.'with* a. 
c.o.mparisoh . of- theit strategies v tor... f oJrikowing- ^fn^t.ructiohs for 
operating a piece .Of Vv elec.trohic equipment .' V' 

-• ' • • r • •. ." '•'• : :\, ■ > •; ; v •• ' ; -\- • . '■' : . 

In .the second; category ' of. work,.', prior ' knowledge /.was 
manipulated, in' 'experiments.- -in« • 'which " subjects';' were vvg-iven 
-inform^t ioh about equipment in /a ^ fixed 'seqiietipe, • .with 'the 
'question being/.how./l-atSr fearriing , was i,nf lUenced' \ by ,>.the 
previously' acquired k^iowle.d^e^. One major topi cf-'in- this/; category 
is .the, role of How 'it t .works -know^e'd^e,, which' is' -the 'knowledge;.; 
that a» . per'-son might. ha vie-' abbut^ "the. 'inte^nai structure »""and 
functioning p.f a j>Iece ' .'of ., Qquip^m?nt. • Y tt?*^ i,s ( , .uno^e'ar •: whether 
•having . --such' knowledge is / "benef icial f . arid- '-if "so* -under,- what 
conditions. ,;it • is.'" A •••.^e'rie's* '-jot'-' ' experimental .studies; . *and 
computer simulation ''mo dell rig efforj' provided' spme;, answers,.. ''•": Sir^ce^ 
.much equipment operation -,1s learned . urid&r . rote learning 
' cQndltdons "a" .second" topi6 ,l was the effects, of .pri pi*, 'knowledge of": 
..procedures «for operatifig' a piece A of ' equdptaeiat . This wo^.k'..,.ar rived' 
.at', a., precise, theoretic} a^.'' \descriptic)'n of khowledge /of procedures 

and trahsfer of 1 -training betweerf procedures. •• • " 

••'•V ' •* ' . • ! \-,\ ' . ' . ■ " . 

' ' ..*' .. \ ' ■ , •••••• t ... ' ," ' v '. ■ 

' . ■ - ' • " prIs'-experimMtal PRIQR .KNOWLEDGB "' " • 

■ Frior'; Knowledge, ahd Learning , f ro.m • Technical P^rose ■ y . 

: sr..,. l^ickgrOund.. '.We* all naye the' a'/trorvg intuition that it should 
•he Easier ;.to iearn frcjm. njaterlai. if one is' already-. faniii-itr* with 
the g«nel?al ; sublet Vfljiatter 'ajf*,ea''.'i However, .there 'have,.-* >1&e.en' very 
f §w'.- atud'iea showing that Vv^iie was^ jirue .'in "'af ocn^inc'ihg \^ay\ and 
in faotr" some' repp-rts .h^ve- s^^esi^d t'he ;op|»Qpite . ' Thi* unol.ear 



*'stat£ of . empirical' Affairs presents . a . ^.raoua theoretical 
••.•problem, because all of . the current, theories ; of ■ corner e hen ajgW:- 
assume that ' <oew. information from .the material is understood :ih 
terms *of ipri or 'knowledge, anU then Uf integrate^ with this priory 
.knowledge. o^B^I/he « avail ability of relevant pr.ior knowledge., 
should have profound effects . : on t the process ,of learning/. 
, information' from^ text vV : The u/iclear . state/ of /.the empirical 
li/terature is probably.. #ue to- the fact that it is >ery diff iou.lt . 
til conduct, a well-controlled/' study in -whi-cjl the amount of true • 
priori knowledge . ! possessed by . subjects is/ •• • the'., variable ; >of. 
. ^interest- Manipulating' .. the amount, of pi* i of .knowledge wi thin, an.' 

'experiment is, not a good, approach ,, because were Is. ample •.reason ' 
« to ' believe. ' from the Experimental Mt.erature that s#ch 
experimentally acquired, knowledge does , /not have >' th'e /. saiii.e 
proper ties*as the true : prior 'knowledge tha ; t ,'sub'j ects hav.fe; before.- 

l - coming -to an experiment". . ■ ". /'V. / . \'v. 

. ■ . '. * . ' . 7 ' 

Appr oach . . The ' approach'.- Cwa$ '■ basfed Q.n v ..the' v. lo^.ic of/ 
quasi-experimental design^'; in Which: * the'/ sampling jpf sub j e^ts -;is 
used to vary* the- amount '. 'Of prior knowledge, and .statist icul # 
met ho(ls,-r mainly m multiple regression,, /'were used , to control 
. nuisance, variables such ' asi-v" general /reading ability'*' 1 word, 
frequency and so. forth. The materials* w<£r'e /several passages that, 
varied widely in. their •• -basic ' content ■ / familiarity ,. both within- 
passages .and \be t we en passages;. ■•■ Thrs; .insured that individual;. 
.. subjects would be almost certain, to vary; Ip,: s t : fce amoiunt of prior 
knowledge that they 'had of the material/-. -'Each' subject was tested; 
to determine which passage : facts' they, aMM^y^^'-J--^.^-'-,,, 

Since ' ' tfu]*e#Vs' can '\v.-chan^.- v'/thelr .^re«i*thg^ behavior- 
substantially af • a, function' of the -^xUng : tasky, $kcee different 
*" reading ' tas Wwere . used . • and ' measure /were collected ..not only of 
the 'amount '"of information • •recalled , /but also, of fWl£ time: spent 
' reading- .or studying the material; /"• Ik '/-one \ : task-,, subjects,., were; 

• 'allowed to study the material for /: as,, -long as: they choe.ey on <a 
self-pacea sentence-.by-sent'fenc^ basi/s,^ ' with* - the ' knowledge that, 
they would' ;be tested for recall* lat^r-. in//a .second readying task|- 
ibg .subjepts knew' that th,ey woflld be tested f Or/ later recall.., but 
were allowed to study each.. Sentence for -"a i^ed" amount. of/tomaV, 

* 'lri the^h'ird' test , •'.subjects d'id....'4i'ot ; Juri*)*-.' ^haVW'*^ would be 
\*- ;#sted <f or'.' 'later .recall, • but:- '• insVfV 1 reaa/t^' fcai^agee* -a . sentence; 

• • at -h timei^h .order to identify, ihfc topic \6f 7 Jjie pas sage y) -All 
s&biects : '.wer©" , 'th.en tested'; for .regal',!.,; ..-.;.;.'. f /f- ' ./•'*'■ . v \- •'• ' / 

, *• »v, u : • • ■ '■■ ■■ ■ > ...■•/■ ••••"•'••.• . * 

.*'' \ ' yp'au itsl'V" A& expected, ' .b^t ' /not p re vlously ; . Remonstrated, = 
; '%heV^" were- .clear' effects - of ^th'e amount of /priory^ntwled^e .. 
' '• However, the , roie : of jpM or . knowledge' depondod . substant ia'Uy on 
.'- the reading task.. v . 'tf-i^xh^ e6ibs'/kne^they>would.':%aV i '3/-to'' .1^.^1.1/^ 
' /mate rial < "'-they ^pentifior^ :1 t iiAe fstudyirig , unfami,! i'ar, .-po.rt i pn^. of 

• ../ the .mai^eri'al'^itian- ^hey-^d ott:. familiar .$q?t?®tia\c r But in", both 
; • oases', t«hey . recallod-^ the..' } ' information; :/ at; ,$frer stme -ley el, 

• re^ardleqa-- of , familiarity, $j)> But . %f ' sub-jeets' " Unaware the 



requirement for later recall, or were limited in the amount of 
time "they could study,, then unfamiliar material was recalled les.s 
than Tain i 1 Dir. . ' " ' # * 

\> ■ ~ ; • ■ \ r. 

The quantitative size of the effects on ^tudy time could be 
accounted for by a simulation J model that was based . on the. 
principle that representations of previously known information do 
not have to <be constructed frofe "scratch", ■ but rather, the: 
previously present representation could simply be "tagged" as 
appearing in a particular passage context. # ^This means that' known 
facts do not have' . to' be subjected to memofi zaf ion' processes to 
the sape 'extent as unknown facts \ v meaning that s-tudy tijme would 
be a function of the amount of unknown information. , 

. ■ The simulation model v was based on standard concepts in 4 
cOmprehensiton theory,, being based *-on . ah augmented transition 
network parser that constructs semantic network representations. 
After each sentenoq was analyzed, the* simulation would compare 
the content of the sentence with the contends of long' term 
memory, and identify'- those portions of" knowledge structure 
that were already present in long term, memory. Then,\ only the 
new 3'tVucture would have to be added to memory. ^ 

: Sign ificance . This ' work provided a demonstration of- a' 
theoretically important effect which had not appeared in the 
literature. It also showed that the effects of prior knowledge 
could be accounted * for using, standard- theoretical concepts 
developed in comprehension research-* One important substantive* 
result is that readfers who are experienced students have mnemonic 
strategies that are powerful enough £o\. deal . with extremely 
unfamiliar material. Another result is that a relatively 'simple 
interpretation of how prior knowledge is used in comprehension is 
viable, as well as more complex w and currently popular, no'tions 
based, on schema theory (e.g., Rumelhart, 1980). % <• * 

■ • *■ 
However, perhaps more 'importantly, this work made it clear 
that how prior knowledge is used * in. comprehension can- be highlyt 
dependent on the task required of the reader..- Standard recall' 
experiments could give' • highly misleading results, because 
subjects can approach the task ■ in a way that can eliminate* 
differences between familiar and unfamiliar, information. 

The original approach planned in this project was taf use 
standard comprehension paradigms to, investigate how passages 
ubout equipment were flfrocessed in tering of the reader 1 8 prior 
knowledge. However, these results strongly suggested that the 
tasks used in such experiments would yield results that were 
either weak, or tied so directly to these .specific tasks that" 
they would not be directly relevant to. the actual tasks involved 
in .operating equipment. For this reason, the /remaining 
experiments in the project always required the subject to engage 



in a .relatively realistic /tasK involving operating a piece of 
equipment , . • " ♦ w * * • . ' 

P ublications > The empirical work , was described, in detail in 
Technical Report Number' 1 1 (Johnson and _ Kieras , , 1 9B2 ) 9 and the 
simulation model was described ' in 4 e "frafl i n Kieras ( 1 9B3 ) The 
empirical work* and the •simulation modeling of the effects 
.appeared in the archival .. publication Johnson and Kieras ( 1 985 ) • 
The methodology used to compare the simulation to the data,, both 
for this study, and' others discussed in this report j was 
described in Kieras (1984a). .Further discussion of . the 
theoretical mechanisms of task effects will' appear in Kieras (in 
preparation) . 

Exper t Kno wledge of E quipment 

Background * While there has bee^ considerable research on 
the nature of expertise, there , has been very little stufcly of 
expertise with regard to equipment, especially actual electronic 
equipment. Compared to other . doraaihs, the domain of equipment, 
has some important ' psychological properties. First of all, 
rather than dealing with abstract concepts, as in "many fields of 
expertise that have been studied, expertise with equipment deals 
with physically concrete - objects. Second J equipment is often 
very ^omplex in terms of .the •different possible levels of 
analysis and types of. information that can be involved. For 
exkmple, knowledge of equipment ranges from the typical colors 
with tyhich the equipment, is .paint ed , all the wstf - to the physical 
principles involved in how the equipment operates. .Third, 
equipment is' something that a person interacts with, and which 
can have its own internal states- and rules of behavior* Thus, 
unlike some ■ forms of expertise which ^ deal, only with a person's' 
.skill, expertise with equipn^ent Involves .not only the', skill of 
operating the equipment, but alao how -the equipment will behave 
in response" -"to what the operator does with , it. Thus knowledge 
about equipment can have some important features that distinguish 
it from other knowledge domains ♦ 

Notice that the domain^of 'electronic devices , is very suited 
for. the exploration * of schema theory (see Rumelhart, 1980; 
Kumelhart & Ortony, 1977)* A schema is aji organized body of 
knowledge that represents' a stereotyped, or a frequently 
ocurring, pattern of events that is used to organize -perceptual 
and memory, processes. For example, device schemas apparently 
conform to very concrete specific physical features of the device 
v and their relationships. Devices themselves have, a very strong 
^hierarchical structure, Bince, the entire device, is made up of 
sub-devices. -Furthermore, each sub-device normally has schematic 
features as well. For example, many electronic devices include 
Bt?me £ort ■ of audio amplifier connected to some sor,t of 
transducer, such as a loud speaker. Such devices almost ' always 
have a cluster, of features . that correspond * to this common 



sub-device. For < example , there will be 'a volume control, quite 
often a tone control, .and a speaker which is normally a circular 
obj-ect behind a perforated grill.. "While details of the placement 
< and ' appearance of these features .will vary, quite often the two 
controls will be located adjacent to' each other, normally 
'clustered with other controls, and the speaker will usually be 
facing toward the user. The Volume and tone controls are very 
rarely separated • from each other, and the speaker almost' never 
appears on the top surface of a device. Because of the' strength 
and concreteness of these patterns,' it would probably be ' much- 
easier to construct a schema theory in this domain than in the 
traditional domain's where the concept, has been used, such as in 
the understanding of storCes^v .-"*-* 



Approach. Both experts and non-experts were used'. in these 
studies. The experts were individuals who had years of 
experience in electronics, many of them being f.drmer military 
electronics, technicians. ;' The ' non-experts were ordinary 
undergraduate and graduate students with no special background. 
"The task was to provide oral , descriptions of a piece of. Equipment, 
which was placed in front of the subject. The' suuject was 
■videotaped, and the descriptions transcribed and analyzed for 
.content.' - The equipment consisted of several .devices, ranging 
from everyday items such as a. tape recorder ,• to very specialised 
equipment. In a follow-up study, ordinary, student subjects wer& 
asked tQ produce, descriptions from memory of several everyday 
pieces of electronic -equipment, « such as 'a television set. The 
subjects were asked, to produce information in several categor ies. 
which were chosen on the basis of the previous experiment 
results. These responses were analyzed for content. 

The value • of the schema concept was explored with' 
small-scale simulation models for how a device would be 
.recognized . in. terms of sche'mas . . . ' 

Resu lts . .The results of the first study' showed that for 
both experts and non-experts, knowledge ' of electronic devices 
classifies naturally into the oategories- of :.. • ( 1 ) the function of 
the device; .(2) the operating procedures; (3) how the device 
works internally; (4) how the- device behaves externally; and (5) 
the power- source- of the device. The . knowledge appeared to be 
•organized in. ter.ms of a hierarchy, in that, the' categories of 
function, operation, and h'ow^-it-works were applied ' recurs ively to 
not just the device as a whole, but. to each of its controls and 
other external features. " . • • 

There'was strong evidence that the knowledge was organized 
in terms' of schemas, in that there were, many cases in which the 
subjects manifested paving definite expectations about device- 
features, and knowledge, of general conventional, 'patterns of 
features. The devices" . were recognized and categorized almost 
immediately by • key patterns -of features. This is exactly the' 
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m fa » 

mechanism that schema organization 4 would entail. Income cases, 
involving' unusual . devices, art incorrect y schema can apparently be 
triggered <by . a subset of ' features, leading - to -serious 
misperceptions of the device as a whole. For qxample 'one expert 
was confused by ? an unusual device that was Actually a form: of 
signal generator, but which.- had its. output* connectors on the 
left-hand, side* of the front panel, »a position customarily used 
"for input connectors 



A surprising result was the pro'minence of 1 ' procedural 
information' jj,s -opposed to how-it-works* information, .which even 
expert 'subjects tended not: to produce. ' Another ^surprise was the 
hi£ih frequency of mention of the power source . of 'the device. , 
FinSally, there were many obvioug features of % the dev ices that 
subjects bftert failed to mention, such* as the fact that an' 
elefc trie clock has hands on a dial numbered from 1 * to 12. 
* 

The second study clarified these t questions by. prompting 
sulfjeets for descriptions from memory ot\ everyday" devices . They* 
wc?»e asked to describe the 'function of each device, the features 
su«h as' dials and indicators which they used to recognize the 
device, and*, in addition, the features' one would expect to see' on 
the device, how the (Tevice was operated, and how the device- works 
iiside. . "There was high agreement * -between subjects about '^he^ 
function,, recognition features, expected . features, an* operating' 
procedures. The surprising prominence of power source 
Information Reappeared in these results, which Suggests that 
)Owct sources, being a common feature of- many different* 
devices, plays an important role in this knowledge 
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The Kb w- it*- works knowledge, was' sketchy** and 
the, first , experiment, although subjects 



inconsistent , as 
■did -produce a 
in the first 
features of a 
concrete , ' the 
or 

' involved 



11 invisible , " 
in the 



■in 

considerably greater quantity than they. did 
experiment. Note that unlike .. the function -and 
device, which are frequently . • experienced and 
how-it-works .knowledge is generally abstract 
being hidden inside the device, and normally not 
routine use of the. device. For complex devices such as' televison 
sets, the' how-it-works , knowledge was less .consistent find 
sketchier ^than for the simpler devices .\ Thus , the lack n of ' 
how-it-works descriptions in the first study was art-if actual , at # 
least to some extent, because, for the simplest devices there* was 
jx reasonable amourwt of such information produced* Perhaps the 
faee-to^f ace. interaction and task derqamds of describing orally a 
presented de,vice biased subjects • against producing much . of this 
knowledge -in the first £tudy* 
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The hypothesis .that devi.ee knowledge is organized in terms 

of schemas' was further confirmed by the fact that the recognition 

•features and expect'ed features were distinct sets, even though 

there was some' overlap* In' terms of schema, theory, certain 
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features, would tr.iggar ( the activation of a schema, .« which .would 
then make, the expecte^d features available. . • : . . 

These results also - contain many of the effects- attributed., to 
;sehemas, such as expectations on the part . of the- per.ceiv'e'r , "and 
confusion wht?n confronted by a stimulus that- does not' quite match 
a schema.- However,; it . is hard. to- define hoW * this 
Characterization could be.'' made -, mor-e precise on the- basis of 
empirical' data; the research would quickly become' a 'joaatter .of 
•simply surveying., the population stereotypes for various -u^eces 
of equipment. While this might be *. useful,' it would not advance, 
the theoretical concepts very far. . k . 

Some simulation modelling of^ schema processes .was dbne. A 
set of production, rules were written that would recursively 

.instantiate .the Schema for a' device , J>y first recognizing' low 
level schemas'for individual clusters of features on the device. 
These sub-sChemas » would then be slot r f i ller.s for higher level 
schemas. a After working its' way -up two or three, levels f ■ the 
schema for the entire device . could' then -be -instantiated . . Working' 

'from a different" tack,- other production rules " could* recognize 
particular .patterns of distinguishing features of a device, and 
directly instantiate the 'corresponding • schema. For' example,, a 
box with an antenna and a turning dial, is .almost certainly a 
radio. While this model was promising,- extending it did not have 
an obvious, direction. It. is hard to say .what any empirical 
consequences of such a model might be, other, than "the obvious 
onesalready described. However, it. is important to note that 
much of schema theory is not; very well developed 'at' the level of' 
rigorous simulation models. Thus , * further work along these lines 
might help firm up the theory. * 
. J 

Signif icance . A major; lesson of^th'ese results is that the 
knowledge that people have about electronic devices is incredibly 
rich' and detailed, encompassing a very ' bread range of kinds of 
information. Perhaps there • has been some tendency for. 
psychologists - and educators to assume that . equipment W.as 
relatively simple, in the sense that there was r relatively little' 
.one needed to know about it in .order ' to. use . -it, *While in some 
sense this .may be true, it is also' clear that 'people- know a very 
large amount of, information of -several different Xypes about 
equi pment .• - 

There ^re many facets of knowledge about equipment' tjhat 
couLd be explored with further research . For J exaraple > from' the 
results it' is clear- that ' famil iar items of equipment ' of ten have 
stereotypical layouts of .the controls, indicators, and so forth. 
One might wonder if this is true of all classes of equipment , "or 
just those that are very common. For example, does the ' extremely ■ 
specialized .equipment used. in the military- also follow 
stereotypical patterns, in its control layout? ..Another related, 
question is whether sterotypical external layouts of equipment' 
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are Jin fact related to the Internal structure of th£ eq\Hpment v 
For example, one intuition is that 'the external features of a 
device that are' most; Closely ^rala^d 'to the purpose of the 
equipmeftt • are . of ten larg6 aficT centrally located. An example is 
that measuring instruments, usually hav'e a rkrge meter, centrally 
placed on' the front panel;. A signal gene^tor usually Kas" a 
large di&l ceptljally posit iorfedV . How will the user 'react terse* 
.signal generator that for design reasonsvhas a small dial located 
in* an of f-cent^r^position? Note that^ modern test, equipment, in 
wtjlich '-'d t igHal/ circuitry* and 'displays are .used^'heavily, o^ten 
G$em- to depart seriously "from previous customs on the placement 
of controls' and indicators. For example", . the front p 4 anel. layout 
of 'service oscilloscopes? has been {fairly standard, because . to a 
great extent because this payout corresponded to the optimum 
arrangement of the* v trad i^ional large ajid -bulky vacuum tube 
circuitry. Newer, more compact circuitry can .be arranged in many 
different ways, thus possibly leading to" front panel arrangeraen' 
that are.. no longer familiar Jo most users- ■• • « r ■ 

\' { P ublications . Theae results are Ascribed in Technioal 
Report flo. 12 (Kie^as, 1902aJ,, ami were discussed in ' a, 
presentation at th.e Joint j'Services Workshop on Artificial 
♦intelligence Applications to Maintenance, whose prpcejedings were 
published ( see Kieras, 19§4b). 

Expertise Effects in 'Following Inst ructions ✓ 

v Backg round ; A very ponua^o<uiask is oum in - which . the user of 
at piece of equipment' follows step-by^step instructions for 
ojler.ati-ng it, Solne^imes '"the goal of^^he user is to learn the 
procedure, but often the user isXin % a stjrictly one-shot 
situation,, meaning "that the user/ is not trying to learn the 
procedure-, ori^ly follow it once. Theoretically, the user must 
obtain specific pieces of procedural knowledge from the 
individual instruction steps, and then immediately execute the 
procedural knowledge. While this, should be a very simple ta'sk, 
it is> clear from everyday^ experience that instructions may" not be 
written well enough to be' followed easily. Furthermore^ 4 the 
experience of the user should play some role in the ability to 
follow instructions oorrectly. \This study was ..designed to answer 
three questions about how people function in a. task involving 
operating a< piecd* of equipment, from" written instructiori$ •> 

The first question concerned the instruction format. „Two 
formats were examined. In -botto- instruction formats, the overall 
; task of the -subject was' to /get the device into a specified 
state; this task was stated . at the beginning ..'of the 
instructions. The first, format was s"fcep-by~step instructions, in 
which each step ooneerned' the setting of a individual control* on 
the 'equipment. Thijr sub j,ety$/Kha'd • to read each step in the 
presented Order, andVwae* expected Afy /carry it out immediately. 
The other instruction format.' was a: ; 'hierarchicai menu. •• Following 
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the' initial task • abatement , the qub-ject was presented with a menu 
of choices, each of which- consisted of a natural "chunk" in the-- 

• operation, of "the device. For example^ if the task was to get a. 
radio 1<uned to a ftfcpecific station, the first menu would contain 
the choices of getting the radio powered up, "getting .the radio 
tuned to the correct? station, and making the. final adjustments' to' 
the radio, controls,. - At each level of the hierarchy, the subject 
gould either attempt to execute the 'task using the information 

v Available at that level, or could choose to get more detail by" 
selecting- one of the choices. This would produce another memi. 

• At the. very bottom ' - of the" 'hierarchy was the same step-by-step 
instruct ion* .as in the .other condition. • Thus, the contrast was 
between following- a linear sequence of steps* to operate the 
controls, or, having the same steps arranged at the. bottom of a 
hierarchy that allowed the subject to read .only to the level 'of 
detail that he. or she desired in order to complete the task, 
'intuition, would hold that this highly organized £orm of 
instructions would, be- superior to' the .linear instructions, 
furthermore, Smith and Goodman (1982), found that ^ similar form 
of hierarchical organization did imprpve performance. • 



The' second ' queqt ion^ was whether there would be substantial 
differences between experts and non-experts in following 
instructions.; It was.. ^Xpected SL that experts would be faster 
overall. However, expertise should also be relative to\ the 
specific de*vice being operated. This was examined by including a 
wide range of electronic device-s, ranging from everyday 'items, 
such as an ordinary portable radio, to' ones famil.iar only to 
experts, such as a dual-trace triggered oscillisaope , to devices 
that would 'be unfamiliar' even to electronics experts, such as a 
laboratory physiological stimulator, or e. unique device. 

The third question concerns ' the nature of -the prior 
^knowledge that ^ibjects have about devices. In the -work 
described above, it was concluded, that knowledge^of devices is 
organized as sohemas, which reflect stereotypical arrangements of 
events. A straightforward •extension of this idea is that 
knowledge, of . operating procedures, for familiar devices should 
also' have a/~Stereo typical pattern. If people's knowledge 
of how to 'operate devices -has . ste reotypical properties, this 
should be reflected in the pattern of menu choices that people 
make in^ the hierarchical instruct ions condition, or a .chunking 
effect in execution time o'f the linear instructions. Also, there^ 
were many cases in the hierarchical instructions condition in\ 
whi ch the device was operated completely from' memory; thftt 
subjects read only the portion^of the instructions that 'stated 
the-oyeral t l task. In- this cas-e. 'there should be schema-basfcd • 
storetHypioality in the sequence of -(operations that people 
performed from memory. ' " • 

App roach , Several de/lces were used, and^Lthere were two 
•type's o : f~< subjects: "• ,^/perts, who .had extemfcve electronic 



experience, and non-experts who. Hid not..- The Instruction type 

.wan between subjects manipulation, but each sUbj ect .operated 

each one of the several cletfices. #'A laboratory corarpute,r was used 

to present the instructions, anjl measure the .time each segment pf 

the instructions was viewed., '.The subject^ 1 behavior was^ 

.videotaped ,in order to .permit "detailed -scoring' of the subjects' 

activities. .''''«> •. , • \ \ 

■ i 

Befor.e beginning the eljqj^jtient , ' the subjects answered a' 
quostionaire in which their, 'familiar i-ty with specif ic.>tems of 
oquipift&nt was assessed- Th,e major dependent variable was'the 
total time , required to ^complete the task. , In the linear, 
instruction condition, the • individual time spent on each step was 
an additional dependent variable. In the menu format condition, 
the jmaj.or variables were the time spent on • each frame of the 
instructions, an-d the choices made in ; moving down the menu 
hierarchy. Finally, the sequence ,of . operating the controls was 
ilso assessed, although this. Only has particular value in the 
menu hierarchy condition, in cases Where very few or no frames i of 
instructions, were read. . Otherwi/se ,■ the controls operated are 
almost cora-ttietely . determined by^Vhat the. presented instruction 
step actually says to do . ^ * • - ' 

R esults . , Contrary to intuition, the hierarchical menu 
format was not - superior overall to the" linear step-by-step 
instruction format. The menu format Was st>perior only if the 
subject was familiar with the type of device,, and. was sometimes 
substantially inferior. For example, while the pVnu condition 
took less than half ^the time- of the . step-by-step cond ition % f or 
operating the radio, in other cases, in which 'the subject. was not 
familiar with the device, the menu condition -aguld ■ be to. 71^ 

slower in total task" time than the step-by-s|^p condition . 

The basic treason for the menu condition being .inferior in 
some cases appealed to be that subjects would often mi-s takingly 
attempt to operate an unfamiliar device on the b^sis of very 
little instructions; this "go it alone" approach .could sometimes 
be d isasterous . For example, expert subjects often attempted to 
operate the physiological stimulator only on the basis Of th.e 
main task statement.. The task req'uired connecting an indicator 
light to the/ 'stimulator* but there -were. ' several possible 
connectors, where the light could be plugged in. Many expert 
subjects plugged the Indicator «Lamp into the vrong ^aok, and then 
•spent a long, time trying to accomplish the task Th#is , the 
intuition thtrt'-the menu hierarchy is "better than the linear 
sequence ' must be strongly qualified by whether the, ua-er is 
familiar, enough with the equipment to take . advantage of \he 
hierarchy, and also whether the subject i's likely to attempt to 
operate the equipment without help* from the instructions when the 
device* is unfamiliar. •' 
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Experts were taster overall than the non-expe rtSEand were 
able to operate equipment with fewer /instructions in^the menu 
conditions • In tertoa cJf total time, experts were about one third* 
.faster thfin non-experts. However,, there s^^re strong' effects of 
specific experience with the device as "well as gen^rCt^xpert ise 
effects. Another major expertise effect was, that experts were* 
mu^h betterra>t executing complicated .physical, activities, such as 
/ plugging in a co*rd, and also at complex % physical activities that 
are fajailiar to experts, such>as zeroing a meter. s Informally, it 
appeared that nonexpert sub j ects of ten would spen^d a lot of time 
fumtfling with' cords arid connectors , whi^le experts seenf to know 



exactly how to perform these activities smoothly and precisely. 
This reeult^^Ls particularly interesting, because it shows , that* 
expertise vrrth electronic equipment h^s other components in 
addition to cognitive factor 6* 

The results showed that' pri\or knowledge played a specif its' 
role in following instructions in the step-by7step condition. 
Examination of. the times to complete individual steps showed that 
a k step that was always read in ,.the menu condition took 
appreciably longer to execute than one that was, never read. 
Assuming t hart the menu choices reflect the familiarity* of the 
procedure chunks, the amount of time taken to complete a step is 
thus a function of its predictability on ^the basis of prior 
knowledge. ,< • . 

Unfortunately the information in the pattern of menu choices 
was very limited. The hope was that the subjects' choice of 
branches of the menu would reflect their knpwledge ' of 
stereotypical portions of- operating the equipment..' This would 
•only be true if the device was familiar to the subject. But, irf 
this case, the subject would need to rfc^d very little of the 
instruction hierarchy. Thus, the ^subjact being familiar with' the 
equipttienst meant that the subject made very few choices that would 
reveal that familiarity. T^ture research along /these lines 
should take this, into account* There were, however, some 
interesting effects in the menu choices. Almost everybody, 
including the non-experts, knew how to get even the expert 
equipment powered up, even if it was unfamiliar. 

"Another interesting . effect is that non-experts co*uld 
JLearn how' to . "perform repeated activities in the course of' 
following the instructions* For example, ' on one device, two 
Indicator lights had to be plugged in one after the other. While 
half of the non-experts read the detailed * instructions for 
pLugging in the first light, very few read> the instructions for 
doing the same to the second light. Similar effects showed up in 
more, complicated situations* The implication 'is , that subjects 
are not simply executing .the instructions as they are' read, and 
then simply forgetting the instruction content ae they proceed to 
the next instruction* Rather, they seem to be able to 
immediately generalize .the content of orie set of instructions and 



(Apply it immediately to a 'similar situation. This suggests that 
subjects' ability to induce and generalize procedures is^very 
rapid and powerful; a similar conclusion was reached in entirely 
different fasks (Kieras and Bovair 1985, Kieras, 1984c). 

An important-aspect of the role of prior i knowledge 
was- revealed .by the sequence of. actions performed by subjects in 
the menu condition, considering those cases in which the, subject 
'operated the equipment successfully using I only" the main task 
statement. It was expected that these sequences of actions would 
show fairly stereotypical patterns. However^ *,the detailed 
atK^lysis of the sequences yielded the conclusion that' there is in 
fact very little stereotypical content in the activity. , The 
initial stages of operating -'.some of - the deVices were fairly 
patterned; for example, all of the subjects plugged in the radio 
before operating any other controls.. Thus,, there is some 
tendency for the power-up . operations to be 'done • prior "to other 
steps. However, there were very few other patterns. 

TWia lack of stereotyped patterns led to the conclusion .that 
subjects operate familiar equipment from" mdmory not by executing 
"canned" procedures but by- problera^solv ing t within- the 
constraints imposed by the nature of, the device. For example, 
many equipment controls have loose- sequential constraints 
on their operation, but these constraints do not predetermine a 
particular sequence of operations. Thus, the subjects made 
many idiosyncratic passes over the controls, and the overall 
state of tne device gradually converges to the desired one. In 
many cases, the number &f control operations performed is 
considerably more than . is' technically required." 

The best characterization of* how a .piece of. equipment 
is operated from memory seems to 4 be that "•people determine what 
constraints need to be satisfied, and then operate the controls 
in a manner that meets the ■ constraints ' and accomplishes the 
task, but does not necessarily follow any, .fixed order. _ Thus, the 
major prediction' of schema theory with regard to how equipment is 
operated, namely, stereotype'd sequences of actions, does .not 
appear to hold.- '• 

« 

• Notice that thja is a task. situation in which subjects" were 
not specifically trained to operate a piece ' of equipment, but 
rather were operating the equipment based 'uport their general 
prior knowledge* A distinction Should be made between what 
people do , when they have a highiy 'automated skill at operating- a 
particular' piece of equipment, a result of, intensive training and 
practice, and the ability to operate equipment in a' more' general 
setting, in whifch each piece of equipment is familiar , ' but not 
highly practiced. Under certain conditions , the strategies 
used by experts may be less effective with unfamiliar equipment 
.than the performance of non-experts who are following strict 
step'-by-step instructions* . 



Kve-n though this f .problera-ao.l.ving -approach is rather/ 
sub-optimal from, u strictly* technical point . o'f view, it is in 
fad, very robust. That is, the .. 'subjects could apply the same' 
approach *to any de.vice.' within a . class ' with which- they were 
familiar. For example, . almost any . electronics expert- would 
operate almost any type, of volt-ohra-railliammeter • They would 
simply recognize which constraints have to be satisfied before 
the desired, measurement could ' "b.e obtained, and would "work 
with the controls, until these constraints were satisfied. It is 
this robustness, of expert knowledge which . is^ particularly . 
valuable^ in operating with - equipment.- In. this pxfyefXmnt/ 'the 
exports could opera-te some of the completely 'hovel ' dev ices* 
without any* instuct ions ,. and do so quite often without any 
serious mistakes or inefficiencies. 

i • * ■ 

' • • «. .. 

Thus, expertise at. operating a variety of equipment does not ■ 
consist of „• having a set of" earthed procedures for operating 
different "devices, but rather of having a set of powerful 
problem-solving heuristics which can ' be applied to devices that 
might be unfamiliar', ,but which may not be very efficient when, 
applied to familiar devices. * 

• Signifi cance . These results are important to the design of 
equipment maintenance documentation, which is usually used, in 
just the' manner explored in this study. The intuition that the 
hierarchical instructions are clearly better than a s-frict linear 
sequence is false; the experience of the' user is critical.- With- 
♦regard to training individuals to become expert users of a broad 
variety of equipment, it should be recognized that teaching 
strict procedures is probably not the appropriate course.. 
Clearly, if the individual is being trained to • operate one piece 
of equipment under stressful conditions, training specific 
operating procedures to the point where they become automatically 
executed, is clearly optimum. However, if the individual is being 
trairted to do maintenance work that might involve a' large variety 
of equipment,. an understanding of the general constraints 
"involved in successful functioning of the equipment . would 
probably be more productive than attempting to teach specific 
operating sequences for each individual piece of equipment.- 

■ \ 

Publications . These results are described,, .in Technical 
Report No. 1 4 (Kieras . Tibbitts,-' & Bovair, 1984), and are also 
cited in Kieras (1984b).. 

EXPERIMENT ALLY.- ACQUIRED PRIOR KNOWLEDGE 

Prior Knoyledge of How a' System Wor ks 

Background. . There # has been a long-standing disagreement 
over whether*' users of ' equipment • should be* fully informed about 
how the equipment works inside. Should the user simply be told 



how to gut the] jti.b.' done*' w'ith'th'e equipment , ^f^ip^lSli- .t.h>.'..\^er . 'b© 
•'/told - 'how -'-the •• '©qiipiiss'rl't^. ubv\f}V - ■ for : exapplsY -tK%.^it r.^\n I n§ : 

".!'nmt«Kin.lrt \*.tot wor'd : '^.proceWors normally- " ;c6w^ : yi^h;. extensive 
*^ diacusaidh 'o'f~how to. ^accompl'^ah Various > : tasK4V wl%.. * the w ; prd.; 

; process or ,. but. ; ! /nOrmalX^ . HaV'e .'. ''' Xi^tle • .&&•/•/ rip. dispj^ftJ'n op. -. 
•^••'i«Sct'iS:t>4cm about/h&w^ the ''Watem .i.t»Vl.f works •v-'-;:\.i "..>•' •••'*'• 

• . ' • • . •„■ .•• ■ - ; -. .<•••- ' • . ' ■ » 

.a. ..On ah : i : . /: ! 6'ther '^hand i ( . there ;.'\are classical results-, ' -in ;; : 
experimental- psychology^ ' that . 'au^gest y.Ky strongly thatf' 
* • . understand Ing,' how -a system* works wou^'d' make ;i*t ' ^e'an,in^ful» , V'anfe: 
y" this would *. greatly. Improve'; a' ,jper'son '.'s- ., ;ab.0 i ty-' to > .IjeaVn the' 
: procedures and to remember '< them later/ ' \ »' *- : : • ; • 

. ' • j ■ ; • • ••• ■ ' 

, " . There have beeri many attempts fin experimental psychology 
to d^rtonstmt'e riust' '.such a beneficial effect of how-it-wo.rks 
" " ..know ledge jfr - the*, contdxt of .sys£e#s'. like , text editors , '.but 
thene* attempts have almost uniformly failed to demonstrate. .t"he.. 
desired effects (e.g., Alexander, 1982V Fqss., Smith t & Ros.^orv, 
1982). The' work done under this project no-t only demonstrated- 
* these effects- in the context of a simple' control panel device, 
' but. also shed considerable light' on '. the conditions under which 
theue effects would appear^ and' provided a theoretically-based 
simulation model that explains the effects. .: 

Appr oach. Subjects were asked to learn' how to* operate a 
simple control panel device that ha'd a few switches, push 
"buttons, and indicator lights.. All of the studies involved 
' comparing two conditions: In the. rote condition,' subjects. 
'.' ' learned how to operate the device strictly by rote, without any 
knowledge of the internal functioning or structure of the device, 
or without any explanation of the' behavior of the device. In the 
model condition, before attempting to learn how to operate the 
device, the subjects learned a "mental model/' in the form of a 
block diagram of the .internal structure of the device, and 
learned how"the internal component's 'of the device were reiated-to 
each - other and to the controls. Performance • in the learning task 
wa^s then ^compared. • ' 

In the first, study, subjects were explicitly trained to 
operate the device in. several* situations. These situations 
corresponded not only -to normal- operating conditions, but also to 
n Ltua'tions • in . which- some internal component to the device was 
malfunctioning, and an alternative procedure -had to be executed 
in order /to compensate for the malfunction. This was intended, to 
simulate the situations involved in., working with real equipment.. 

After subjects had learned all the procedures, they Ve-re 
then gfven a series -of retention tests/ The major variables were 
the time* taken to stud£ the .how-it-works explanation, the time 
taken to complete the training phase, the execution speed and 
accuracy during the retention tests,' and 'certain qualitative 
features Of retention, such as whether the recalled procedure was 



:ui ' improvement, upon? thotfe .'that hud. ''been taught. '" That ; is , . tfre 
■ jws true ted. procedure^" h'adboen "deigned so that some,«;Of , ; them 'were 
'<l-u%o, . 'In'eff i«{«ht r .." V -..-it- 'th^e .'suVj.ec t kryety w : ; to ,' 'correctly 
.inter/]ar>;t ^th^ indicator .lights,, .the procedure, could be 
.cpntuderab^y '^h ; or>en'eU .... • ' •'...' "• ' •"' '.. 

'"'• v ry'.' '' : V, •. ';• ..'•••.'' ■■ .. ■*' \. •' ' ''•'•••< 

'...•'in.'the ^second, a/id third st.udi-.es. f-- the subjects, inferred how 
;to : ppevate ; t W;. die v? £c6y • '- rather • t han> : . be"i /e^pl to it ly. trained to 
■operate- \ .it .' ' Eft • this, ^ procedure., the;' Vsulbj.ec.ts' were given the 
device,; told what the ;desi red' c goal- state was, and - then were free 
.to ' try/* to - operate the. controls to" "arMve at t hat goal state.. 
A l though, operator^ of • 'equipment .are - not .normally put in this" 
situation,,; this' procedure proved to-be a simple and effective .way 
Of determining . .and • ex-ami 'ning the effects of havi,ng *the 
•h0%-it-yor.ks^knowle3gei,, . ' ' > v t . ' v 

,.;. "•';•, Results., In' the. first experiment', in. whic,h the procedures 
'.'Were |e%li<3itXy ►trained* -the • group who had studied how the system 
.work^'learrted ^-the procedures faste°r, retained • them, better , and 
executed ''them faster ,, even after one* week.. A typical -effect size 
was -a 20$ improvement. Q£ special . interest is, that the model 
group made tfhe procedures more efficient far more- often than the 
rote grOup.'V; This m$ans - ■ -that not, only was the how-it-works 
knowledge .'producing a general improvement in • performance, 
but also va qualitative improvement in subjects; ability to deal 
intelligently with the device. ' The time taken to lea'rn the 
mental I model was roughly' the same as the sayings in training 
time, but, notice that the model materials were not optimized. 
Thus, wVth no 'additional training time penalty, the model 
subjects \ere able to deal with the device much better. 

, A simple explanation for these results is that knowing' hoV 
the system worked made it more. . ""meaningful . " However,- this 
explanation is not .detailed eripugh. • A more precise hypothesis is 
that the how-it-works kno'wledge allowed the subjects ,to infer the 
procedures, which would give the subject two independent means of 
executing a procedure correctly. That is, the direct' rote memory 
for the procedure failed, the subject could reconstruct the 
procedure based on 'inference from knowledge of ho.w the device 
worked. In .'some cases , this' inferred procedure could in faot.be 
more efficient than the instructed one. 

/ 

The second and third experiments confirmed this inference 
hypothesis* The subjects were asked to infer the procedures 
rather than learn them from- explicit training. The second 
experiment simply compared a -group trained on the model with a 
r-ote group in the procedure inference paradigm* The results were 
quite simple; the model grqup could infer, the optimum procedures 
on /the first try, whereas the rote group took several tries to 
arrive alt the same procedures by sophisticated, but* limited, 
trial atjd ' error approaches . Think-out-loud protocols showed, 
quite clearly that . the model group was basing -their inferences on 



knowledge, 'of . hoy. the "ays tern .worked, whereas the rote 'group was 
•basing .theih irtfe rences. on the superficial details of- how the 
•.device"''-'- looked/. aj}.d behaved, such as the fact, that there were 
'subtly, .. pirvembhic relationships- between the labels on various 
controls aVid . 'indicators. Another interesting result from the 
thl- n k~o\l t-, ^Olid protocols is that the rote group subjects tended 
to.' view -:the .device- as . unreliable^' and caprici-ous in its behavior , 
and«.^©n -suspected 'duplicity on the .part of V the experimenter. 
Tft,es.e ( affe.pt -lad en reactions suggest- that much of* what tfe think 
o^as^computer anxiety." could in fact be due to the coghitive 
.problem, of not being able to ..explain or predict ^ow a -system is 
behav ing. ' • t - r 

The third experiment was ' intended t© determine what aspects 
of Knowledge, of. how the device worked ,wa.s important. 'The 
previous experiments had supplied information- about how/ the/ 
system worked in terms, of a fantasy explanation based on the 
"Star -Trek" television series. ^Namely, the' control ^panel was 
described as • being' the control panel for a' "phaser bank" . abojard 
the "starship Enterprise". While this fantasy certainly 
mot ivated, and interested subjects, there is an obvdoiys concern 
about- whether the effects produced were due to general properties 
of this -fantasy. Also, . the mat erial . "included- not only "a 
description of the internal components of the system and their 
relations to each, other t but also some discussion of' the 
fictitious principles of physics involved. These principles may 
Mve provided some organizing structure, and thus might have, 
produced the effects. - 

. •. ' / 
The third study' was designed to demonstrate 'more clearly the 
nature of <tvhe critical information in the mental model, based on 
the idea that a good mental model supports inference oft the 
procedures^ The two factors compared k were whether the 
how-.it-works material had the fantasy and fictitious principle 
Content or not, and whether the material provided the system 

-topo logy, , which is information about how the components and 
controls are connected to each other". Such materials included a 
block diagram and discussion about the actual controls on the 
control panel, and how they were connected to the actual internal 
components of the system. The materials in the no-topology 
information with fantasy-principle content also presented a block, 
diagram, bu^ this diagram did not include any of the actual 
cpntrols of the. , system} rather it corresponded* to : an idealized 

*general description of. how systems of this sort ', worked, rather 
than the specif ic system that; the subject was dealing with". '• 

* The condition corresponding to no fantasy-principle content 
arfti no system topology information- was ^essentially the- same as 
the rote condition ,o| the earlier' studies. The condition with 
the fantasy-principle^ content and the topology informat ion' ,was 
the... same- as the previous model conditions. . .The 'critical 
comparison is whether the fantasy and principle content provides 
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any performance facilitation, or whethe^ the topology information 
is the critical content. 

The results were Very clear. The/ fantasy-principle content 
provided no facilitation at all; the critical information was the . 
system topology Information. The /fantasy-principle 'condition 
with no topology information provided /a general' discussion of the 
principles* of how systems of this typ^ worked, but this did not 
allow subjects , to irifey the actual procedures . needed to operate, 
th-e^device. Rather, it was, critical to know how the controls '' 
- related'* to each other and 4 to the components. 

'Thus-.tlty< general conclusion./ can <be stated in ^erms of ,a 
criterion for when, how- it-wOrks knowledge. -ftill>be, of, value to the 
^uaer: How-^t-works knowledge will be of- value d*ftly~y it is 
Specific enough tp allow the user [ to infer the exact operatic / 
p roc ecfures . 'Thus , the earlier atftempts to demonstrate positive ' / 
FelaefTts of understanding" how a w.oijd processor wqrks probably did 
not provide, knowledge, that was 'specif ic enough. On the other 
Hand, only some * of the information about ht>w .a system works 
should ^e important; frying to understand . technical detail trtiat % • 
is not "needed in order' to be j^able to, infer the operating, 
procedures is simply a waste-'of tiflie\ * ' 

This leads to a second criterion for^when how-it-works 
knowledge should 1 be provided to the user of a piece of 
equipment: How-it-works knowledge should only be 'provided if it • 
is actually necessary or advantageous to the user to be able to 
infer th e procedure^ /rather than learn them by rote . Notice that 
the ordinary telephone system is ejo easy to learn by rote that it 
is doubtful whether being able to. infer how to operate it from, 
knowledge of the switching .mechariisms ; would ' be ^ -of any value. 
Similar arguments can be / made ' for everyday systems like the 
automobile j ■ , ^ . 

»» % • * i 

In the 'cage of word processors, many of the commands that 
are 'involved in operating a word processor are either determined 
arbitrarily* by the person who wrote the software, or are- obvious . 
to the user in terms of the text editing task itself. For 
example, no explanation is necessary for . why pressing the 
up-arrow cursor key causes ' the cursor to move up. Likewise, no 
amount of explanation of the principles of, . computing or wor.d 
processor design will explain why "EUN" is. the keystroke sequence 
that will. exit the editor;' this was simply an arbitrary decision' 
on the part of the designer. 

Depending on the specifics of the design of the ♦system, 
there may indeed be aspects of the how-it-works knowledge that is 
important for the user, to krlow. however , ♦ this knowledge should 
be very specific.,, and severely limited, in technical detail. 
'It is a problem, for future research to determine whether this 



knowledge* has ' characteristics s that - would allow it to be 
determined on an a/priori bas-is. ■ 

To explore these hypotheses theoretically, a simulation 
model was constructed » f op how . procedures could be inferred from 
knowledge of the system topology. The simulation model has a 
declarative and a procedural component. The declarative, 
component" is a proposit ional representation of the block diagram,, 
or topological description,, of. the. system. The content ' of the 
block diagram, and the' explanation that accompanied it, was 
basically the power flow connections through the system/ That 
is, the diagram started at the main. power source and went through 
various infernal components and "Switches until it ■ arrived at the. 
final component of the system which required the power., The 
procedural component is a' set of about 50 production rules which 
operate on the declarative representation. These production 
rules generate a plan for operating the device, and then execute 
the plan. -If the plan fails \'to produce, the desired result, the 
rules attempt to determine the problem in the "system that caused 
the failure, and then attempt to devise a new pl-an. 

The . plan is constructed by finding a path "through the block 
diagram of the system that routes pow.er from the source to the 
desired 'point. - The plan consists, of a list of the control 
settings that will establ i.sh the " route . In-order to devise the 
plan, the .production rules essentially simulated the internal, 
state changes of ' the ;..dev ice . This corresponds; to a popular 
notion of : the role of mental models as 'allowing- the person to 
simulate internally the states of .the ^external world. ..Thus, the. 
rules modelled. • the flow of .power, through the system', and made 
simple inferences about . the conditions of individual components 
based. upon the states of the indicator lights. 

Although rather simple, this simulation model represents a 
potentially broad and important class of mental ' models , namely ,- 
any system' in' which some cpmmodity, such as energy- or 
. information, -is routed from, one point to the next, through 
discrete and all-or-none components. Furthermore, models 
of this, class have potential significance other than as 
simulation models f or * cognitive . processing. They bear a strong 
resemblance to certain problems now being attacked in artificial 
intelligence^ in which it is desired to troubleshoot -or analyze 
the/ operation of a system based on a . description of its internal 
s trueture . . . 

\ 

The processing' in the simulation model was compared in 
'considerable detail to the response times of individual actions 
by- subjects in the ' topology information' conditions of .third 
experiment. The 'basic question in this comparison was" whether 
the simulation and human subjects performed »,he ir in l «: reru:'.- 
processes at the same , points in the N 'sequence of actions 
performed, given the cases where-, .subjects performed the same 



,sequen<«:e factions as the simulation; • If. so, the relative 
amouWt of *time required for the inferences should be- accounted 
for by the model. " ' ' . * 

l\i was found that a reasonable . portion of the variance in 
.the .response times could be accounted for 'py the simulation 
model, supporting 'the explanation for' the role of how-it-wor.ks 
knowledge. The simulation was based . on the principle of 
inferring procedures from a logically minimum • required amotfnt of 
knowledge of ( how the system worked, namely, its- topology, along 
with a few simple and general principles for. how power flows from 
one Doint to the next in- such a system. A matter for farther 
resejir.ch is. whether it is -possible to formally characterise which' 
portions of a device'description. are' logically required for this 
form* of inference. *■ * 

Sig nificance . This #ork bears very directly upon a. basic 
issue irT the preparation of training materials'- and of 
documentation for equipment.. There" has been some controversy for 
some time about the role of training, in basic electronics theory 
in the training_ maintenance personnel. For example, Bond and 
Towne (1979) report that\ a common experience is that standard 
training in electronicsr 'theory is ..of little -or no value in 
troubleshooting even complex equipment.' '.' . ' • 

r 

rradltional electronics training deals with very general 
'principles, which although important, may. only ■ rarely explain the 
behavior of a piece of equipment at the level of analysis 
required for troubleshooting and repair.- P©r example, much 
electronic - repair of complex systems' ds done by identifying a 
defective module and replacing it*.. The logic of identifying the 
defective module usually involves reasoning based on tracing "the 
power or signal flow through a set of interconnected modules. 
With complex systems ^.th is reasoning may 'i.n- fact be quite subtle, 
but -it simply does not 7 involve basic electronics theory, , such as 
Ohm's law, or the. details of transistor functioning. Rather, the 
logic of identifying. a defective module is likely to be specific 
to the behavior Of the modules and the. topology of the system 
being repaired; only when one is troubleshooting- at the 
individual discrete . component level wi.ii the more basic 
electronics theory become important. . 

Continuation of this research, and-sin attempt to/ apply it 
more directly to training situations, potentially can result in 
much more efficient and { effective training approaches. Also/ 
potentially more effective ..equipment documentation could be 
prepared by ensur ing. that . the critical how-it-works information 
required is 4 prominent, • and not obBcured* by unimportant 
information. It is not known at this time whether documentation 
in fact provides this critical system topology^inf ormat ion'^in *tn 
easily used way; this would be a topic for further research. 
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Publlca tiySB . The ' details ' of the experimental work is 
described in Technical' Reports No's . 13. (Kieras &Bovair, 198;>)- 
and 15 (Kieras, 1 9JB4c ) , and the s.im'ulat ic-n model and its 
comparison- to data are described in detail in Technical Report 
o. 15- (Kieras, 1984c).' A condensed . presentation • of the 
xperim'ental 'results appeared irf .Kietas and Bovair (1 984) • A 
pkper based on Kieras "'(1984c), concerning the simulation model 
aid the .comparison to . data, has .been submitted to Cognitive 
IJcYLence, and word on acceptance . is .expected within a few months.^ 

*~ . V- 

This work has been" described in several conference and 
colloquium. talks V The .work was presented at the. 1983 
Psyehonomics Society meetings, in a colloquium series on applied' 
cognitive psychology at the University of ' Michigan in December 
1983 A and in a colloquium at Bell , Labs in February of 1984^ .Many 
reprint requests have also been received and responded to. 
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Back ground . In recent^ years 
theory M ... the nature and acquisition of cognitive skill 
(And*rsonY 1982). According to. this theory, people have -both 
d ^S.\^ r JiLil e - and procedural ' knowledge . Declarative knowledge *ls 
ITnowledge^f • fact«, whereas procedural, knowledge is knowledge of. 
how to do things. Thus, in "the context • of ^operating equipment, •* 
the job oflthe learner is to acquire the/knowledge of< how {bo 
actually operate the equipment; which is/ procedural knowledge . 
Knowledge o\ how the' equipment works, as discussed^ above , is 
declarative knowledge. ' ... '. 

'The theorW goes" beyond this simple distinction, however', to \ 
propose -specific representations for both declarative and \ 
procedural knowledge. In-line with established cognitive theory, 
declarative knowledge \b represented as a* semantic network. 
'Procedural knowledge is represented as a set of pro duction 
rules. A production rule is in the form: 

\lP (condition) THEN (action).' , 

A production rule \consists ' of a condition and an action; if the 
condition is satisfied, then the rule is "fired," and the action 
is performed. A sat of production rules consists ^simply of a 
large set of duch rules , with - no built-in constraints upon the 
order in which the \rules' may • be executed* Rather the order in 
which- the ruies .fire is specified by th.e conditions and action^ 
The conditions can teat for both external events * ^such sfs outsider 
stimuli, or internal conditions such as the state of the semantic- 
network representation \ or the contents of a working memory. Thj 
actions' can both modify\ the external situation by means of overt 
responses, or can modiifor the- state of memory. Thus, a piece of 
procedural knowledge consists of a set of production rules whose 
conditions and actions cause the .rules to be fired in the correct 
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»rdor v to produce the proper sequence of oVert actions, 
Anderson's work .has, focuased on -developing principle's of learning 
that describe how an initial ..pet of ^production' rules can become , 
more compact and off iq lent as learning proceeds." This theory has' 
been able to explain many of the -important and classical results 
'in' learning, suoh as the exact mathematical* shape of the learning 
curve . V* , 

In the' theory of cognitive skill, "the initial set of 
production rules *for a particular skill is assumed to be derived 
from . declarative knowledge that is the- original- input to the 
system. That is, . when first, learning a skill, the learner' would 
■acquire, -a body/- of declarative knowledge that" provides the 
specifications for the' skill to be learned. These specifications 
would be interpreted- by some general problem solving process, 
itself represented as production rul.e,s, and as a by-product 'of 
the activity of these, general \rule.s, , specif ic^ rules ' for the 
.particular skill will be formed. 

■ • Quite often In learning to operate • equipment, tyhe' learner 
gots tTie initial specifications for the skill in the 'form of. 
written step-by-step' instructions. In terms of the theory, what 
the- learner must do is to' derive .'a Correct set of production 
rules from the content of these instructions., Presumably, the 
written instructions would be comprehended by mechanisms similar 
to those already proposed ^in current theories of reading 
comprehension (e.g. Kieras, 1982b, 1983). These mechanisms wouid 
result in the learner -having a declarative representation of the 
procedure available in memory' immediately after reading the 
instructions. A general set "of instruct ion-following "processes , 
a pre-existing set of production rules, would then interpret this 
representation and carry out the correct procedure. Again,"as a 
by-product of ttye • activity of these general procedures, the 
specific production rules for the particular procedure would then 
be formed . « • • » " . . 

Anderson's work' has focused almost completely upon, the 
processes that occur once the" correct production rules have 
already been formed. This work focused 'on the process by which 
the written instructions" were, translated into production rules. 
Thus, ,this work complements Anderson's, in that it focuses on the 
very initial stages of learning a skill from written material. 

Approach. . The rote condition* of the first experiment on 
how-it-works knowledge described above yielded data on learning a 
series of procedures by rote. The time' required to learn the 
individual procedures varied over a 'very large range, and 
appeared to be a -function of the order in which the procedures 
wore learned, After translating the procedures into production" 
rules, it appeared that the excursions in training time could be 
accounted for by a simple transfer of training hypothesis. This 
hypothesis .held .that in learning a procedure, production rule's 
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that had b&eri learned* . jtVyJtft pr^ev'iou^ j)roce^iure could be,- 
'transferred into* the. rVp^^nt.,^t fOr^the' . hey "procedure if the' 

rule was either identical ^''Of.^VeV^ a^mifljr toVx tne rule required 
'for the .new procedure. M J;,.Jjh^-5A:.'/.t^ ' '"ijime 'feq.uir'eci to<learnj^ 

pro6edure would ' be" r joaos.^y*\ a, fu-nutijion ' ot' /the nurobe'r, of n 



production rules ' rptyuUr^f by /the 'procedure . " This simple rule 
could; account for' raoa.t;. .pf./the variance in -tfoe rote learning data 
from the •hfrw-f/t-'works'- i^ot^-'edge ^<pQ.rj^'e<Vit 'dgsori'bed above. 



The worta, described in , . t this sectiun., .was. (1 ,theri' undertaken, to 
provide a mo/e comprehensive ^. test of s„' t\{t$«: simple transfer of 
training tlfeory. The subject's, learned *p'rbc0#u res for operating a 
simple control panel /device , which was the. same one used. in . the. 
studies. of how-it-.workg> knowledge . .describe a above. However, -in 
this work*, subjects were;' not provided Nan y information about 'how 
the system worked.' .Rather,, 'they learned ' how , to operate the 
device strictly~by rote. .The training' was done by explicitly 
listing the individual steps in- ihe, procedures , and having the 
subjects- study .'these instruction^,' ''followed by V attempting to 
reproduce the procedure frpm ^memory^/ ; 'T.hi;s process was repeated 
until the v subject^ had learned'the procedure . 

A simulation mod el* was constructed, to rigorously simulate, 
the transfer process, v The production rule representations for 
each of the 10 procedures used were then put through the transfer 
"simulation in* vapfous training orders.' A set .of three training" 
orders was then chosen and., used jin the experiment, A. set of 
step-by-step instructions was devised for each procedure that had 
the. property, that >aach individual ' sentence > stating an 
instruction step' corresponded \nery well to the contents of one 
production rule. 

' ■ 

In the experiment, sub jects t first read through, the step by 
step instructions, and then attempted to • execute the procedure 
from memory. If they made . .a mistake, th'ey were cycled back 
through' the instructions. They repeated this alternation between 
reading the instructions and 'trying to execute' the procedure 
until they successfully executed the procedure three times in a 
row. Then they went on to the next procedure in the specific 
training order. The variables'- of interest were- the total, time 
taken to,' learn a procedure, , the time spent reading each 
individual instruction step, the accuracy of execution of each 

"individual step in" the procedure during training, and the speed 
and accuracy of retention fn each step in the procedures in a 

•final, test for memory in the procedures. 

• ; . ' r ** 

The transfer- simulation model ' was 1 used to make rigorous * 

1 Priori predictions of the : number of new and transferred 

pr^Tuct ron rules in each procedure as . a unction* of^the training 

order. It was expected that these predictions could account, for 

a substantia^ portion of the variance in training times, as was 

suggested by v the preliminary analysis of the data disoussed ' 
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•above* By examining the relationship .between the r.eading times 
of Individual, instruction 3teps with*, the accuracy of execution of 
tjie corresponding step in the procedure, it should be possible, to 
essentially track £he acquisition of individual production 
fules. In this' way, information could . be obtained pn the Very 
Initial^ stages of acquiring a procedure < from written 
instructions. ' . • ■ *. 

Results . As expected, ,the * analysis of the procedures in 
'terms of the. transfer hypothesis was* able^ to account for a 
considerable proportion of the. variance in .training times. The 
number of new production- rules required by a . procedure* was the 
single \ most 'important of 'the possible /predictor , variables 
considered, and alone could account for 69$ of tfte variance in 
training' times,* arid was a better, predictor of training time, on a 
single procedure than a subject's own mean training ; time . 

A detailed regression ( analysts irevealed that there were 
other effects involved in procedural Iparhing as well* Qf 
special interest 'is. an apparent "overload 1 ' effect, in which, 
tt certain complex procedure was the first to be learned , 'and 
considerably more, training time was required than would , be 
.predicted on the basis of the amount- of new production rule 
information involved. Thus., while training time on the whole is 
very closely related to the number of new"production rules, there 
are other aspects of training order' Trhi&h can b,e, very important . 
These aspects can be <jl*ea # rly identified by applying the 
production rule analysis 

A, matter for furtfS3|^>esearch is exploring the nature of 
some. of the acMitional .effect^ , and . testing the general! ty of the 
transfer theory, and whether the production* rule analysis is as 
powerful in a variety of different task domains as it is with a 
simple control panelT^ Notice that Poison . c^ad Kieras (1985) have 
applied a similar analysis to the learning of a word, processing 
system, and found, sin/ilar predictive power *of the production rule^ 
analyaip* Thus*, it appears* that the production rule analysis* 
is very ' general ,-'*but further - research is needed to explore its 
limits , . " . . , 

• ♦ 

A detailed , analysis of the reading times for the individual 
instruction steps showed that people essentially cease to spend 
•time on' instruction steps once the corresponding steps in the 
HB^ocedure has^ been mastered. What is particularly surprising, 
however, is that this effect only ^ appears for instruction steps 
that correspond to' new production rules as defined by the 
transfer theory. Procedure steps thaj correspond to previously 
learned production rules do not show such a sharp decline in 
reading time. In other words, upon the. very first reading of a 
procedure, subjects pan distinguish between those, instruction 
steps that correspond to production rules they already know, and 
those that. correspond to rules that have to be learned. 
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Instruction steps that are already known are read for very little •' 
time), right from the' outset, whereas steps .for new rules are read . 
and 3tudled until they are mastered, after which the reading time 
drops down to the same as that for steps ..already known. • V 

• What- this pattern. of reading time effects suggests is that 
the coirect. execution of a procedure, depends on when a correct 
declarative representatio-n of the procedure has been formed, and 
not upon -when a production, rule „ representation for it has 
been formed. That is, the pattern of reading time v results 
appears to resemble what would be expected from .. powerful 
comprehension processes that- 'can compares and manipulate 
declarative representations. It does not appear to" be easily 
explained by the learning rules proposed by Anderson (1982) that 
are de/ined in terras of operations upon procedural knowledge. 
This means that the initial s.tage*£pl learning from written text 
have more to do with comprehension processes than originally 
believed. . ' . 

A preliminary analysis of. the retention data' suggests that 
the production rule analysis may also be very powerful in 
explaining the ' details of retentioh of procedures. Many of the 
errors made in. recalling procedures could, "be accounted for by 
interference between two of ( the production rules in the 
procedures. Such -rules had very, similar Conditions, differing in 
literally only one bit of information, but different actions., one 
being the correct action, and the other, producing an incorrect 
action. 'About 95^ of the errors in. recall were due to the 
similar incorrect rule being fired" instead of t>a-. correct one . 

However, the severity of .the interference of the incorrect 
rule was strongly" .related to classical variables from 
interference theorty. For example, the .amount of practice 
with the two rules, and their ordering during training, were jjLery 
important,. This shows that the traditional degree of learning 
and proactive versus retroactive interference considerations are 
at work in procedure.- retention.^ More- importantly, the exact 
details of how these classical ' variables show up in recall 
can apparently be easily characterized in terras of the production 
rule analysis. A matter for future research is, to clarify these 
e-ffects ' further, ana construct » a rigorous model that -predicts 
•where these, retention interf erence\ef f eots will occur. - 

Si gnificance 4 . This work is unique and unprecedented in that; 
it is the first time that important quantitative features of the 
learning process could be accounted for with . su<5h, power and - 
.precision 'by a completely a priori" analysis ♦ As^jghown by the 
PolAon and 'Kieras work, this analysis also appeals 9 to be. very 
general, -. but additional research is • needed to; confirm this 
claim. If this theory of learning and •. transfer can be 
successfully extended, it provides a very powerful analytic tool 
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for investigating and improving -the efficiency of training 
materials and training. sequences. 

As applied in a related project being conducted by Kieras 
and Poison under sponsorship of the IBM corporation, . the analysis 
■ can be used to evaluate proposed designs for user interfaces' of, 
computer 4 systems (see ftieras & Poison, in press; Poison & Kieras, 
1985). That. is,, a high-quality user interface is one. in which 
there is relatively little in the way of procedural knowledge 
that has to be learned in order, to operate, the system,, and in 
which there will be strong positive transfer of production rules 
from one procedure to the next, corresponding to a "consistent" 
user interface. Thus, ;^he* practical significance of this work 
could be very large; further research will tell 'whether, this 
potential is real. 

On the purely theoretical front, this work has played an 
important role in clarifying the nature of procedural knowledge, 
and- how it is acquired from written- material* Together with 
other work being -conducted- under 0NR sponsorship, such as 
Anderson's, there should soon be a comprehensive body of theory 
dire.ctly related to training issues at a level of precision and 
' practical value that was simply not available before. 
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Publications 



This work- has been described 



t} _ in Technical. 

Report. No. 16 (Kieras & Bovair, 1985), and in a paper at the 1984 
Cognitive Science Society Meetings. Related. work from the kieras 
and Poison project has been -described in Kieras and. Poison (in" 
resb) , and in conference presentations , by Poison and Kieras 
1985). A journal article based on Technical Report No. 1.6 will 
be submitted within the K&ext few months. 

«, •• ' SUMMARY / 

The work in this project has .made important contributions to 
the understanding of the role of prior knowledge in operating 
devices from written instructions. In terms of experimental 
methodology, some usefui conclusions ' can be stated. First, 
readers have specialized strategies- for dealing with unfamiliar 
material, which' means that. many traditiona.1 prose recall 
paradigms should be used with caution in the investigation of 
prior knowledge.. The direction of this project, had to be"- 
changed because as originally proposed, it would have .relied 
heavily on standard comprehension paradigms and , thus was 
vulnerable to producing misleading ' conclusions about prior 
knowledge. Secpnd, in experiments investigating mental" models , 
or other forms of prior knowledge, careful attention should *b.e 
paid to the relationship between the knowledge being supplied to 
the subject, and. the exact tasks that the subjept is expected to 
perform. Previous research in this area has not considered this 
relationship ' in .enough detail, leading to many failed experiments 
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and confusion- over the role of prior knowledge', especially in th.e 
how-»it~worka domain. 

On the theoretical front, ' this , work has- continued to 
demonstrate the power and. effectiveness of rigorous theoretical 
analysis of the sort that s oan be represented in -a simulation 
model. 4 The: ' effects of prior knowledge in Uprose memory 
situations, how-it-works knowledge utilization, and transfer of 
training, can 'all be explained by Simulation models in a way that 
is theoretically precise,- arid in many cases quantitative an<f 
empirically powerful. On the whole, the results .support what ia 
perhaps becoming the consensus model of cognitive architecture, 
namely the ACT class Of theories described in Anderson's most 
rece.nt textbook, The Archit ecture of Cognition (1983). ^ 

The practical significance of these results is substantial, 
but will require further research to. fully realize. With regard 
to instructional materials, both for immediate execution and 
long term learning, there are several important conclusions 
regarding the arrangement, .sequence, and content of the 
material. If the • conclusions from this research are confirmed by 
'further research, it will, be possible to make very precise 
decisions about what should 'be included in both training 
materials and operating instructions for equipment.. Good choices 
could be made . about what level of detail of haw-it-works 
knowledge should be included in training materials and equipment 
doeuraehtation. The exact' Sequence and content of procedural 
instructions can be chosen • with great precision. The overall 
arrangement and content of instructions for immediate execution 
can be chosen with regard to the expertise and knowledge of the 
users of the instructions. Thus, these results of this project 
. clearly prov-ide a good foundation for future . applied -research. 
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